{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Weight-Space Symmetry\n",
    "\n",
    "According to Bishop,\n",
    "> Multiple distinct choices for the weight vector $\\bf w$ will give rise to the same mapping functioins from inputs to outputs\n",
    "\n",
    "This will be useful for Bayesian model comparison"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from numpy.random import randn, seed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Consider, for example, a two layered neural network with $\\tanh$ activation function, input vector $x$ and a set of weights ${\\bf w} = ({\\bf w}^{(1)}, {\\bf w}^{(2)})$.\n",
    "\n",
    "$$\n",
    "    x = \\begin{bmatrix}\n",
    "    x_1 \\\\\n",
    "    x_2\n",
    "    \\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "    {\\bf w}^{(1)} = \\begin{bmatrix}\n",
    "    w_{11}^{(1)} & w_{12}^{(1)}\\\\\n",
    "    w_{21}^{(1)} & w_{22}^{(1)}\\\\\n",
    "    w_{31}^{(1)} & w_{32}^{(1)}\n",
    "    \\end{bmatrix}\n",
    "$$\n",
    "\n",
    "$$\n",
    "    {\\bf w}^{(2)} = \\begin{bmatrix}\n",
    "    w_{11}^{(1)} & w_{12}^{(1)} & w_{13}^{(1)} \\\\\n",
    "    \\end{bmatrix}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Tanh symmetry"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed(314)\n",
    "W1 = randn(3, 2)\n",
    "W2 = randn(1, 3)\n",
    "x = np.array([[1, 2]]).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feedforwarding the network, we obtain the hidden units for the hidden layer as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 = W1 @ x\n",
    "z1 = np.tanh(a1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we obtain the value of the output layer as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.73209407]]\n"
     ]
    }
   ],
   "source": [
    "a2 = W2 @ z1\n",
    "y = np.tanh(a2)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The weight-space symmetry works as follows: we can obtain the same result for $\\hat y$ by _changing_ the signs of a particular group of weights. \n",
    "\n",
    "Considering our two-layered neural network. Letting ${\\bf \\hat w}^{(1)} = -{\\bf w}^{(1)}$ and ${\\bf \\hat w}^{(2)} = -{\\bf w}^{(2)}$, we obtain the same output value for different values in the network."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "W1p = -W1\n",
    "W2p = -W2\n",
    "x = np.array([[1, 2]]).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 = W1p @ x\n",
    "z1 = np.tanh(a1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.73209407]]\n"
     ]
    }
   ],
   "source": [
    "a2 = W2p @ z1\n",
    "y = np.tanh(a2)\n",
    "print(y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> Interchanging the values of all the weights [...] leading beath into and out of hidden hidden unit with the corresponding weights [...] associated with a different hidden unit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [],
   "source": [
    "a1 = W1[::-1] @ x\n",
    "z1 = np.tanh(a1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[-0.73209407]]\n"
     ]
    }
   ],
   "source": [
    "a2 = W2[:, ::-1] @ z1\n",
    "y = np.tanh(a2)\n",
    "print(y)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
